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Abstract Body 


Background & Context for the Study 

As education policy in the U.S. focuses increasingly on the quality of classroom instruction, 
there is a growing need to identify valid constructs and develop reliable tools that measure 
student opportunities to learn academic knowledge and skills (Strong, 2011). Given the diversity 
of classroom inputs with the potential to impact student learning — e.g., content area, curriculum, 
resources, student autonomy, classroom management, assessment practices, teacher modeling, 
time on task, affective teacher-student relationships, and student practice opportunities — we face 
a series of important questions: How do these features of instruction relate to one another? How 
can they be measured? Which instructional practices should be emphasized for different 
purposes associated with student learning, and for which students? 

Although there has been extensive research on the discreet features of classroom 
environments that contribute to student learning in elementary school classrooms, particularly in 
the context of early literacy instruction (e.g., Connor et ah, 2009; Curby et ah, 2011; Gersten et 
ah, 2005; La Paro et ah, 2011), these features have received less attention in middle grade 
literacy classrooms. Identifying the instructional practices that support adolescent literacy 
development is critical for several reasons. First, many middle school students do not read at a 
level required for success in higher education and the workplace (ACT, 2006; Kamil et al., 2008; 
NCES, 2011). Second, the reading demands placed on all students increase dramatically in the 
middle grades, as reading proficiency becomes the gateway to accessing content across all 
academic areas (Common Core State Standards, 2011). Third, efforts to synthesize effects of 
adolescent literacy research largely reveal that existing interventions are not effective for 
increasing student achievement outside of controlled studies (e.g., Enhanced Reading 
Opportunities and Striving Readers studies), indicating a need to provide intense interventions 
for struggling adolescent readers (e.g., Kamil, et al., 2008; Scammacca et al., 2007; Biancarosa 
& Snow, 2004). To identify practices that can increase adolescent literacy achievement, and in 
turn, design interventions and promote instruction that may be more effective in the middle 
grades, we contend it is important to study discreet features of adolescent literacy instruction 
implemented in middle school classrooms. 

A common means of examining enacted instructional practices is direct observation. 
Standardized observation procedures are effective measurement tactics in instructional 
intervention research, and can improve our understanding of instructional and environmental 
factors that are causally related to student achievement (Kane & Staiger, 2012; Lipsey & 
Cordray, 2000). In this study we used two instruments to document features of adolescent 
literacy instruction and examine mediation effects between initial reading skill and student 
reading achievement: (a) the Ratings of Classroom Management and Instructional Supports 
(RCMIS: Doabler & Nelson-Walker, 2009), a measure of general instructional quality; and (b) 
the Classroom Observations of Student-Teacher Interactions (COSTI: Smolkowski & Gunn, 
2012), a measure of contextual features of the classroom environment (e.g., number of minutes 
of instruction, content focus of instruction) and the rate of instructional interactions between 
teachers and students. These two instruments were used to examine instructional practices in 
Reading Intervention (RI) classrooms in schools participating in a longitudinal, multi-component 
project to evaluate middle school literacy instruction, student engagement, and data based 
decision-making. 
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Study Purpose & Research Questions 

The purpose of this study is to examine the relative contributions of two different classroom 
observation instruments in an evaluation of classroom instruction in a comprehensive reading 
intervention program designed to support struggling readers. We explore whether instructional 
practices observed using each instrument mediate the effect of initial skill on student reading 
achievement, as a means for identifying practices that contribute to adolescent literacy 
development. Our specific research questions are: (a) Does general quality of instruction as 
measured by the RCMIS predict student reading achievement, above and beyond initial skill; (b) 
Does the rate of student-teacher interactions as measured by the COSTI predict student reading 
achievement, above and beyond initial skill; and (c) Does the RCMIS or the COSTI predict 
student reading achievement better than the other, after accounting for initial skill? 

Setting & Participants 

This study was conducted in 25 middle schools in six school districts in the Pacific 
Northwest. Trained staff observed 213 seventh grade Reading Intervention (RI) classrooms 
between one and four times across the 2011-2012 school year (M= 2.36). Altogether, 502 
reading intervention observations were conducted in 2011-2012. Participants were 1,267 seventh 
grade intervention students and 124 RI instructors. 

Description of the Intervention 

The Middle School Intervention Project (MSIP) is a five-year (2010-2015), longitudinal 
study to evaluate a comprehensive intervention program for middle school students with reading 
difficulties in six districts in the Pacific Northwest. The intervention includes three components: 
(a) Reading Intervention, (b) School Engagement Intervention, and (c) Data Based Decision 
Making. Although the present study focuses only on the first component, it is important to note 
that these components, as a package, comprise a targeted, Tier 2 intervention that supplements 
established school-wide, universal Tier 1 reading and behavior support systems. Additionally, 
each of the intervention components are operationalized by participating districts; that is, MSIP 
is a partnership between researchers and participating districts, established to evaluate districts’ 
existing instructional practices as they relate to providing comprehensive support to students who 
struggle in the area of reading. 

Participating schools combine published intervention programs such as Language! (Greene, 
2004), Read 180 (Scholastic, 2006), and Corrective Reading (SRA, 1999) with other 
instructional approaches to deliver literacy interventions designed to (a) improve student reading 
outcomes on the state standardized reading assessment, and (b) enable students to meet reading 
demands in content-area classes (e.g., science, social studies). RIs are generally provided as a 
supplement to English Language Arts (ELA) instruction, and focus on critical reading content 
(e.g., word reading instruction, reading fluency, vocabulary, and comprehension strategy 
instruction) (Kamil et ah, 2008; National Reading Panel, 2000; Pressley, 2000). RI classes are 
somewhat smaller than ELA or content-area classes, and meet for a minimum of 200 minutes per 
week. All RI instructors are trained in the reading programs used and in procedures of effective 
instruction for struggling students. 

In addition, participating schools use test data (e.g., standardized reading assessments; 
progress monitoring measures) and other sources of evidence to determine which students 
receive RI and when they should exit the intervention. All students that receive RI also receive 
an engagement intervention (El). Many schools implement formal engagement programs (e.g., 
Check In, Check Out; Check and Connect), while others use infonnal school-determined 
interventions that involve regular, personalized interaction with the same adult). 
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Research Design 

A critical component of the MSIP project is that students are screened into the targeted Tier 2 
intervention components based on 6th grade summative state assessment reading scores (Kamil 
et ah, 2008; Scammacca et al, 2007). Schools are permitted to choose their own cut score, based 
on a rank order list of composite z-scores for each student. Composite scores are comprised of 
the average of (a) a z-score for student performance on a brief measure of oral reading fluency 
(using the sample mean score as the comparison), and (b) a z-score for student performance on 
the state reading assessment (using the state average score as the comparison mean). Each school 
identifies a cut score on the basis of available resources and the number of students they have the 
capacity to serve. All students who score below the school’s identified cut score are screened 
into the comprehensive intervention and considered treatment students. Students who score 
above the school’s selected cut score are considered comparison students. 

To account for systemic variance associated with instructional practices and the assignment 
of students to classes (e.g., students may participate in multiple RI classrooms in a single day or 
across the year), we used a multi-level model nesting classrooms within students within schools 
by including weights for instructional variables of interest, individualized for each student based 
on the dosage of instruction they received from each classroom. 

Measures. We used two measures to assess the diversity of classroom inputs (e.g., 
instructional content area, instructional structure, teacher modeling, teacher feedback, time on 
task, affective teacher-student relationships, student practice opportunities, classroom 
management, and the learning environment) and predict their impact on student learning. 

RCMIS. The RCMIS is an 1 1-item Likert rating system targeting features of instructional 
quality (classroom management, delivery of instruction, and the learning environment). 
Observers use a 4-point scale to rate the overall quality of each item (Doabler & Nelson-Walker, 
2009). Predictive validity coefficients of reading and mathematics outcomes range from .26 to 
42. In the present study, the RCMIS was modified for use in middle school classrooms by 
replacing items targeting teacher-directed features of interactions with student-initiated 
approaches to learning and adding two items targeting student engagement (for a total of 13). 

COSTI. The COSTI is designed to capture the duration, content domain(s), group 
structure(s), and rate of interactions between teachers and students (e.g., teacher models, 
feedback, student practice) during instruction. Studies indicate the COSTI can be used reliably to 
predict early literacy and math achievement (Doabler et al., in press; Fien, et al., 2012; Nelson- 
Walker et al., 2012; Smolkoski & Gunn, 2012). In our previous work with the COSTI, predictive 
validity coefficients with reading and mathematics outcomes have ranged from .25 to .55. 

Student outcome measures. Student outcomes were assessed using scores from the state 
standardized reading achievement assessment and a measure of oral reading fluency (ORF). The 
state standardized reading assessment was an untimed, computer administered measure of 
student skill in comprehending text. The ORF measure was a one -minute, individually 
administered measure of student skill in accurately and fluently reading connected text. 

Data Collection and Analysis 

Observation procedures. Classroom observations using the COSTI and RCMIS were 
conducted in all RI classrooms in the fall, winter, and spring of the 2011-12 academic year. All 
observers (25 data collectors and 10 research staff) had experience participating in research 
studies and/or working in schools. Prior to each observation round, observers received training 
consisting of: (a) a review of content, (b) new content, (c) coding practice, and (d) reliability 
documentation. In total, observers received more than 24 hours of training across the year. 
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At the end of each training session, data collectors watched a video of classroom instruction 
and completed the RCMIS and COSTI. Observers’ ratings were compared to the ratings of the 
observation coordinator and an inter-rater agreement index was calculated. In the fall, observers 
were also required to meet an inter-rater agreement criterion of .85 or better on all measures 
during a field checkout before observing independently in classrooms. Data collectors who did 
not meet criteria participated in additional training activities until they met criteria. Average field 
checkout reliability for the RCMIS was above .95 and for the COSTI was above .92. Reliability 
data were also collected for approximately 20% of all observations in each round to document 
maintenance of inter-rater agreement between trained observers. Efforts were made to pair 
observers (a) with each of the other observers assigned to their district(s), and (b) across regions 
for half of all reliability observations (i.e., cross-district observations). Average inter-rater 
agreement for the RCMIS was above .94 and for the COSTI was above .90. 

Student assessment procedures. Student outcome measures were collected by district and 
project staff. Districts provided student scores on the state assessment in spring 2011 and spring 
2012. Student perfonnance on the state assessment in spring 2011 was used to create the z-score 
composite for intervention assignment and as a measure of initial reading skill, while student 
perfonnance on the state assessment in spring 2012 served as the primary outcome measure. 
Trained staff collected ORF data in the fall of 201 1 and spring of 2012. Spring 2011 assessment 
scores were used to create the z-score composite for intervention assignment. 

Analysis procedures, HLM software (version 6.08; Raudenbush et al., 2004) was used to 
conduct a logical sequence of model testing proceeding from more-to-less constrained parameter 
estimates and to test hypotheses associated with each research question. 

Study Results, 

We found that the quantity of teacher instructional behaviors and student responses, as 
measured by the COSTI, did not predict student achievement gains on a standardized measure of 
reading comprehension. However, the effect of these behaviors varied significantly between 
schools. On the other hand, we found that overall instructional quality, as measured by RCMIS, 
significantly predicted student achievement across schools. In addition, we found that the 
RCMIS appears to be a better predictor than the COSTI when use in middle school RI 
classrooms. A complete correlation matrix and descriptive statistics are provided in Appendix B, 
Table 1. Results of the HLM analyses are reported in Appendix B, Table 2. 

Conclusions 

Initial findings from this study indicate that high quality instruction matters above and 
beyond the quantity of instruction students receive. Preliminary findings suggest the RCMIS can 
be used efficiently to document the quality of instruction provided to students in the middle 
grades. In addition, there appear to be unmeasured features of some intervention classes and 
schools that moderate the relationships between instructional factors and outcomes. Additional 
analyses are needed to explore relations between student-teacher interactions captured by the 
COSTI and student outcomes to determine whether the COSTI may be used to predict 
meaningful differences in middle grade instruction across RI classrooms. 

Overall, these findings provide initial evidence for the types of observational tools that can 
be used to distinguish among the diversity of classroom inputs and how such inputs impact 
student learning. However, these measures may be limited in their ability to identify quality of 
instruction within a particular content area (e.g., reading comprehension). Future research is 
discussed in terms of developing new observational tools to identify specific instruction within 
the content area of reading comprehension and its impact on student learning. 
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Appendix B. Tables and Figures 


Table 1 


Instructional Features Measured by the Ratings of Classroom Management and Instructional 
Supports (RCMIS) and Classroom Observations of Student-Teacher Interactions (COSTI): 
Correlations and Descriptive Statistics (N = 1267) 


Variables 

1 2 

3 

4 M 

SD 

Min 

Max 

1 . Comprehension 
pretest 

- 


218.47 

6.18 

200 

239 

2. Comprehension 
post test 

.596*** - 


224.53 

6.97 

202 

248 

3. Instructional 
Quality 

-.030 .018 

— 

42.23 

5.85 

0 

51.33 

4. Teacher 
Behaviors 

-.205*** -.156*** 

290*** 

292.18 

188.26 

0 

1225.35 

5. Student 
Behaviors 


284 *** 

.971*** 196.29 

130.78 

0 

878.97 


* p < .05, ** p < .01, *** p < .001 
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Table 2 


Fixed Effects for Final School-Level Model 


Estimate of fixed effects 

Variable 

P 

SE 

df 

t 

Intercept p 0 





Intercept Y 0 o 

224.89 

0.45 

24 

495.22*** 

Comprehension Slope Pi 





Intercept Y i 0 

0.67 

0.03 

24 

20.24*** 

Instructional Quality Slope P 2 





Intercept Y 2 o 

0.08 

0.04 

24 

2 . 12 * 

Teacher Behaviors Slope P 3 





Intercept Y 30 

- 0.001 

0.004 

24 

-0.25 

Student Behaviors Slope P 4 





Intercept Y 40 

-0.006 

0.006 

24 

- 1.11 


Estimate of random effects 



Random effect 

Residual variance 

SD 

df 

Chi-square 


component 




Mean p 0 , 

4.49 

2.12 

18 

55.08*** 

Comprehension Slope pi/ 

0.01 

0.10 

18 

31.88* 

Instructional Quality Slope p 2 y 

0.01 

0.11 

18 

32.12* 

Teacher Behaviors Slope p 3/ - 

0.0002 

0.01 

18 

33.97* 

Student Behaviors Slope p 4/ - 

0.0003 

0.02 

18 

32.46* 

Level 1 fa 

25.44 

5.04 




Note: Percent of variance explained demonstrates amount of variation attributed to between 
schools and within schools. 


* p < .05, ** p < .01, ***/?< .001 
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